Sandbox6
Ratemall Corporation uses the Automated Ratings System (ARS) for promotion, award and assignment recommendations, as well as for counseling and performance improvement. Each employee has a page within the system that is accessible via a Web browser. When visiting the ARS Web site, an employee sees his/her own page. If the employee is a supervisor, he/she can navigate to the page of any directly subordinate employee. And the supervisor's own supervisor, who is called the "manager," can also view the pages for the supervisor's direct subordinates. Home Page Let's assume that you are a new employee named Jonathan Studeman. The very first time you visit the ARS, your page might look like this. |} Monthly Rating Requirement Each month the ARS selects the names to be rated that month from a list of persons that the employee is required to rate, called the "Core List," and also from a list of persons that the employee is merely eligible to rate, called the "Extended List." The Core List always contains at least five names, and the two lists, combined, total 20 names. To be eligible for promotion or award, an employee must rate five employees each calendar month. If, at the end of the month, an employee has not done so, the ARS will require that more be rated the next month, and the employee regains eligibility when the deficit is made up. The Core List The supervisor and the employee must agree on a list of no fewer than five names for the Core List. These names can be changed at any time, so long as the supervisor and the employee agree, but must at all times include the supervisor, the supervisor's other direct subordinates (those that have regular contact with the employee), and the employee's own immediate subordinates if the employee is a supervisor. If that number is fewer than five, the supervisor and employee will need to select others with whom the employee has regular professional contact in order to make up the difference. (The manager can and should be included if the manager has regular professional contact with the employee.) At any time, the supervisor or the employee can make whatever changes to the Core List seem desirable or necessary, as when someone on the list changes jobs. If the supervisor adds or removes a name, a new proposed list is submitted to the employee for approval, and if the employee adds or removes a name, a new proposed list is submitted to the supervisor for approval. This continues until a new list proposed by one of them is approved by the other without changes. If never changed, the core list must be re-approved by both at least once per year. The ARS itself generates the Extended List, using names selected from: :* Core Lists of other employees; :* email aliases associated with the employee's office; and, :* organizational charts. The Core List and Extended List are displayed side-by-side. To add a name to the Core List, a name is simply dragged there from the Extended List or searched for in a drop-down Global List of all employees. To remove a name from the Core List, the name is selected and dragged into the Extended List or into the Global List. Ratings Five rating factors must be addressed to submit a rating. :* The option of "No opinion" is not available for persons on the rater's Core List. :* If "No opinion" is selected for any factor, the rater will not receive credit for rating that person. However, the rater will then be asked to rate another person selected from the Extended List, so that the quota of five ratings per month can be fulfilled. ARS Report For every employee, the ARS generates and continually updates a 3-part report that is based on all the ratings submitted by all employees. Access Valid The ARS report is hidden until and unless it is deemed by the ARS to be "valid," and then it is made available only to authorized viewers. To deem a report as valid, the ARS requires that ratings by 10-or-more credible others (see Credibility Assessment, below) for at least three rating factors (including Trustworthiness) reflect a consensus of opinion. A consensus is achieved when the standard deviationStandard deviation is a statistical measure of how widely or narrowly a range of values spreads from the mean (average) value. A small standard deviation indicates that the values are grouped tightly around the mean. A large standard deviation indicates that the values are spread out more evenly across the range of possible values. of the ratings for a factor is less than one. If there is consensus on some of the rating factors but no consensus on others, the ARS can generate a partial report that identifies which factors are excluded. But if there is no consensus on Trustworthiness or there is no consensus on 3-or-more of the rating factors, the report is deemed not valid and remains inaccessible. Pre-authorized Only the employee and the immediate supervisorThe supervisor's pre-authorized access applies only to the Promotion and Awards Report, not to the Assignments, Self-Assessment, and Disciplinary Report. See: Part II: How the Employee was Rated by Others. have continuous, pre-authorized access to a report. The members of the Promotion and Awards Board also have access to the report, but only when the Board has received a recommendation for promotion or award. The Board loses its access after a determination is made. Requested The employee may, from time to time, grant 10-day access to other persons, and should normally do so when applying for a new assignment or when recommended for a new assignment. A prospective new supervisor, or a manager deciding on a assignment, will typically request access to the report. An access request is generated whenever a person who is not pre-authorized to view a report attempts to access the report. The person who attempts such access is asked to either select the purpose from a short list or type the purpose into a text box. The employee is then automatically notified and is required to explicitly grant or deny access. If the request is being made for one of the standard purposes on the list, the employee is required to state a reason if denying the request. Part I: Credibility Assessment The report begins with an assessment of how well or poorly the employee has rated other employees. This is based on a statistical analysis of the ratings submitted by the employee in comparison with the ratings of the same persons by other employees, as well as with the ARS credibility assessments of all employees. If, for a given person and a given factor, the rating submitted by the employee differs from the mean rating by more than one and by more than the standard deviation, that rating is flagged as "questionable." In other words, if a rating is outside the range of the standard deviation but is still only "off" by one (e.g., below average versus average), it is nonetheless accepted as at least "credible." And, if a rating is "off" by more than one but is still within the range of the standard deviation, that rating, too, is still accepted as credible. The only ratings flagged as questionable are those that differ from the average rating by more than one and that lie outside the range of the standard deviation. Individual flags are not revealed in the report. Instead, the system looks at the ratio of "questionable" to "credible" ratings and compares it with the system-wide average. * If the employee has a ratio that is high to a statistically significant degree, the report says, "Less than credible. This employee has submitted a significantly high number of ratings there are deemed not credible." * If the employee has a ratio that does not differ from the corporate average in a statistically significant way, the ARS reports that the employee's participation in the system has been "Credible." Part II: How the Employee was Rated by Others The report next shows an assessment of the employee for each of the five factors, as well as an overall assessment, based on how he was rated by others. Promotion and Awards Report If the report is being used for to support a recommendation for an award or a promotion, the report shows only three possible assessments for each factor and four possible assessments in the overall rating: This is based on the mean rating for each factor for which there was a consensus (standard deviation of less than 1), which excludes any ratings that were not credible (were more than one position off the mean rating and were outside the range of the standard deviation). If there is no consensus for a factor, the rating is excluded from the report and, in its place, the report states "No consensus."The ARS is designed to avoid and remedy "No consensus" situations by bringing in more names and systematically shifting names in and out of an employee's extended list. . : NOTE: For awards and promotions, decision-makers focus on the higher ratings, in order to choose from among the best. If the "inferior" and "among the worst" ratings were routinely revealed to too many people who do not need to see them, the employee might suffer from prejudiced assessments in the future (negative reinforcement), which would both hurt the employee and undermine the integrity of the ARS. Therefore, in these award and promotion reports, "acceptable" serves to cover the entire range of lower ratings for the individual factors, masking the lowest ratings. The ARS still uses the lowest ratings in calculating the overall assessment. Assignments, Self-Assessment, and Disciplinary Report But if the report is being used to support a recommendation for disciplinary action, or for an assignment decision, or by the employee for self-assessment purposes, more detail is provided. In the case of disciplinary action, the disciplinary Board must first make a decision to "take up" the recommendation. This means that the Board has decided to take the recommendation seriously enough to inform the employee and conduct a formal investigation. At that point, the Board must notify the employee, and then the ARS Assignments, Self-Assessment, and Disciplinary Report becomes available to the Board members. In the case of assignment decisions, the manager must request access to the Report, which the employee may refuse. Examples An ARS Promotion and Awards Report might look like this for an employee named Jane Doe. Those "Acceptable" ratings might well be masking a low rating in one or two factors, but it would not be anything very serious because Jane's overall rating is "Superior." But now let's look at John Sample's report. John has not submitted a sufficient number of credible ratings, but there is not much difference in the individual rating factors compared to Jane. However, in John's case, those "Acceptable" ratings must be masking something very, very serious because his overall rating is "Inferior." John, himself, knows exactly what he has to improve upon, because he has already seen his own Assignments, Self-Assessment, and Disciplinary Report. If we, too, were able to see that report, we would know, as John does, exactly what his problem areas are: Clearly, John does not have good work habits, does not work well with others, and is perceived as being untrustworthy. The Promotion and Awards Board members do not need such detailed information since they are only deciding whether to reward John, not how to improve him. : But, can "Among the worst" truly be classified as an "Acceptable" rating in the context of such a Board? Actually, Yes! Remember, promotions and awards are recommended, typically with the supervisor's and the manager's endorsement (and without which the recommendations are unlikely to be approved). Supervisors and managers not only know their employees as well or better than anyone, but they will have seen the more detailed reports already (during initial assignment), and were entitled to request access to the detailed reports before submitting recommendations. If, to them, the ARS "Among the worst" rating was acceptable, then it should be acceptable to the Board. In cases where recommendations come without management endorsement, the Board can and should investigate further. Assignment decisions, however, and disciplinary actions, do call for such detail. When someone requests access to John's report citing a re-assignment decision as the reason, it is this detailed report that they will see if John agrees. If he does not agree, he is required to state a reason. The manager can draw inferences from the refusal and the stated reason. In the case of disciplinary action, the Disciplinary Board can see the report even if John objects, as long as he is notified. Part III: The ARS Recommendation The report concludes by making a recommendation to the Board or to other decision makers. This is the shortest section of the report, and consists of just a single sentence, such as this: To arrive at this final ARS recommendation, points are awarded for the credibility assessment from Part I and for the overall rating reflected in Part II, and then simple addition produces the result. As shown, every employee who is in good standing within the ARS and has an overall rating of Average is at least eligible to be considered for an award, a promotion or a challenging assignment.Even an Inferior, Unworthy employee can still be considered by a board or an assignments manager. Unworthy does not equate to ineligible. The only employees who are completely ineligible for award or promotion are those who have not rated other employees as the system requires. See:Good Standing, above. * But to be deemed worthy of consideration, an average employee must have participated in the ARS in a credible manner. * Similarly, every superior employee will automatically be recommended unless his/her participation has been less than credible. * And, likewise, every excellent employee will be strongly recommended, assuming that the employee has performed credibly within the system. All of this guarantees a very strong incentive for employees to do their ratings very carefully and without bias, while removing any incentive to engage in bias. For one thing, sloppy ratings harm the employee who submits them by significantly lowering his or her chances of being promoted or awarded. For another, not only do biased ratings have the exact same effect, they also neither benefit nor harm the persons toward whom or against whom the bias is directed! For example, if I give a friend a superior rating because he is a friend, that rating will be discounted by the system (and counted against my own credibility) and will do him no good if it varies significantly from the consensus rating. (And it will obviously make him no better off if, by chance, it happens to be in agreement with the consensus!) Therefore, I know, going in, that I cannot possibly help my friends — nor harm those who are not my friends — by submitting biased ratings; I can only harm myself. And what if I fear my friend will ask me how I rated him? Isn't such fear an incentive to bias? Hardly. Ratings are meant to be confidential, and simply reminding my friend of that fact is a good way to impress upon him that I am a reliable employee. He will likely soon be rating me for honesty, integrity and reliability, and will appreciate having a sense of where the consensus on that may lie! So I would not lie to him. I would tell him truly that ratings are meant to be confidential. And I would know that whatever I say, the ARS system itself will never reveal, to anyone, how anyone has rated anyone else. All specific ratings are hidden. Only composite results are ever reported. So there is no reason to fear being asked how I rated someone, and every reason to take such a question as an opportunity to display integrity. If my "friend" rates me lower for having refused to confide in him, he will only be hurting himself. Recommendation Forms Supervisors recommend, managers endorse, and a corporate Board decides on promotions and awards. Forms for promotion or award recommendation include a checkbox for the manager's endorsement of the recommendation. If the option "not endorsed" is checked, the manager must provide a brief explanation. However, a recommendation can come also from a person who is not the supervisor. In that case, the recommendation still normally goes to the manager, who will normally obtain the supervisor's decision to endorse or not-endorse before sending it along to the Board. But a manager, or supervisor, or anyone, may choose to send a recommendation to the Board without an endorsement, though that is very rare and unlikely to be successful. To do this, the person submitting the recommendation must check the "Not endorsed" box for the supervisor and/or manager and must provide an explanation as to why the recommendation is being submitted without a normal endorsement. The "Endorsed" box can be checked only by the authorized person (by the supervisor or by the manager), and will be digitally signed. But the "Not endorsed" box, which will also be digitally signed, can be checked by anyone. Again, normally, the supervisor or the manager is the one who would normally check these boxes and provide any required explanation. He/she then sends the recommendation "up" if endorsed (up to the manager or up to the Board) or "back" if not endorsed (back to the person from whom the recommendation came). If a not-endorsed recommendation "comes back," the originator can still decide to forward it to the Board, so long as the "Not endorsed" box(es) have been checked and digitally signed, and so long as explanations have been provided. A similar form exists for recommending disciplinary action. The title of the form is different, and the form is routed to a separate, Disciplinary Board. However, everything else is the same. Typically, a supervisor recommends and a manager endorses, but anyone can recommend, and anyone can submit the form with or without endorsements, though the same "Not endorsed" box(es) must be checked, and explanation(s) provided. Footnotes